Newton-type Distributed Optimization A. Lower Bounds for One-shot Parameter Averaging

Before providing the proof details, let us first describe the high-level intuition of our construction. Roughly speaking, oneshot averaging works well when the bias of the predictor returned by each machine (as a random vector in Euclidean space, based on the sampled training data) is much smaller than the variance. Since each such predictor is based on independent data, averaging m such predictors reduces the variance by a factor of m, leading to good guarantees. However, averaging has no effect on the bias, so this method is ineffectual when the bias dominates the variance. The construction below shows that when the strong convexity parameter is small, this can indeed happen.